Papers with linguistic probing tasks
LINSPECTOR WEB: A Multilingual Probing Suite for Word Representations (D19-3)
Copied to clipboard
| Challenge: | LINSPECTOR WEB is an open source multilingual inspector to analyze word embeddings. |
| Approach: | They propose to use LINSPECTOR WEB to analyze word embeddings in 28 languages. |
| Outcome: | The system performs 16 simple linguistic probing tasks for a diverse set of 28 languages. |
Information Aggregation for Multi-Head Attention with Routing-by-Agreement (N19-1)
Copied to clipboard
| Challenge: | Existing studies focus on extracting informative or distinct partial-representations from different subspaces, while few studies have paid attention to the aggregation of the extracted partial-Representations. |
| Approach: | They propose to use a routing-by-agreement algorithm to improve multi-head attention by iteratively updating the proportion of how much a part should be assigned to a whole based on agreement between parts and wholes. |
| Outcome: | The proposed algorithm improves the information aggregation for multi-head attention over the standard linear transformation on linguistic probing and machine translation tasks. |
Frequency Balanced Datasets Lead to Better Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evidence that high-frequency tokens in pretraining data might bias learning, causing undesired effects, is not clear. |
| Approach: | They propose a sampling algorithm that iteratively assesses token frequencies and removes sentences that contain still high-frequency tokens, resulting in a balanced dataset. |
| Outcome: | The proposed method reduces the amount of pre-training data required for training attention-based transformer language models by up to three times. |
SSA: Improving Performance With a Better Scoring Function (2026.acl-long)
Copied to clipboard
| Challenge: | Despite the success of in-context learning, recent studies have identified systematic limitations in its generalization behavior. |
| Approach: | They propose a new attention scoring function that mitigates failures in transformer models . they use Scaled Signed Averaging to train the scoring function instead of Softmax . |
| Outcome: | The proposed scoring function outperforms transformer models with Softmax on NLP benchmarks and linguistic probing tasks. |